{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`Learn the Basics <intro.html>`_ ||\n",
    "**Quickstart** ||\n",
    "`Tensors <tensorqs_tutorial.html>`_ ||\n",
    "`Datasets & DataLoaders <data_tutorial.html>`_ ||\n",
    "`Transforms <transforms_tutorial.html>`_ ||\n",
    "`Build Model <buildmodel_tutorial.html>`_ ||\n",
    "`Autograd <autogradqs_tutorial.html>`_ ||\n",
    "`Optimization <optimization_tutorial.html>`_ ||\n",
    "`Save & Load Model <saveloadrun_tutorial.html>`_\n",
    "\n",
    "Quickstart\n",
    "===================\n",
    "This section runs through the API for common tasks in machine learning. Refer to the links in each section to dive deeper.\n",
    "\n",
    "Working with data\n",
    "-----------------\n",
    "PyTorch has two `primitives to work with data <https://pytorch.org/docs/stable/data.html>`_:\n",
    "``torch.utils.data.DataLoader`` and ``torch.utils.data.Dataset``.\n",
    "``Dataset`` stores the samples and their corresponding labels, and ``DataLoader`` wraps an iterable around\n",
    "the ``Dataset``.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import torch\r\n",
    "from torch import nn\r\n",
    "from torch.utils.data import DataLoader\r\n",
    "from torchvision import datasets\r\n",
    "from torchvision.transforms import ToTensor, Lambda, Compose\r\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch offers domain-specific libraries such as `TorchText <https://pytorch.org/text/stable/index.html>`_,\n",
    "`TorchVision <https://pytorch.org/vision/stable/index.html>`_, and `TorchAudio <https://pytorch.org/audio/stable/index.html>`_,\n",
    "all of which include datasets. For this tutorial, we  will be using a TorchVision dataset.\n",
    "\n",
    "The ``torchvision.datasets`` module contains ``Dataset`` objects for many real-world vision data like\n",
    "CIFAR, COCO (`full list here <https://pytorch.org/vision/stable/datasets.html>`_). In this tutorial, we\n",
    "use the FashionMNIST dataset. Every TorchVision ``Dataset`` includes two arguments: ``transform`` and\n",
    "``target_transform`` to modify the samples and labels respectively.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\CreatorFan\\AppData\\Roaming\\Python\\Python38\\site-packages\\torchvision\\datasets\\mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  ..\\torch\\csrc\\utils\\tensor_numpy.cpp:180.)\n",
      "  return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)\n"
     ]
    }
   ],
   "source": [
    "# Download training data from open datasets.\r\n",
    "training_data = datasets.FashionMNIST(\r\n",
    "    root=\"data\",\r\n",
    "    train=True,\r\n",
    "    download=True,\r\n",
    "    transform=ToTensor(),\r\n",
    ")\r\n",
    "\r\n",
    "# Download test data from open datasets.\r\n",
    "test_data = datasets.FashionMNIST(\r\n",
    "    root=\"data\",\r\n",
    "    train=False,\r\n",
    "    download=True,\r\n",
    "    transform=ToTensor(),\r\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pass the ``Dataset`` as an argument to ``DataLoader``. This wraps an iterable over our dataset, and supports\n",
    "automatic batching, sampling, shuffling and multiprocess data loading. Here we define a batch size of 64, i.e. each element\n",
    "in the dataloader iterable will return a batch of 64 features and labels.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X [N, C, H, W]:  torch.Size([64, 1, 28, 28])\n",
      "Shape of y:  torch.Size([64]) torch.int64\n"
     ]
    }
   ],
   "source": [
    "batch_size = 64\r\n",
    "\r\n",
    "# Create data loaders.\r\n",
    "train_dataloader = DataLoader(training_data, batch_size=batch_size)\r\n",
    "test_dataloader = DataLoader(test_data, batch_size=batch_size)\r\n",
    "\r\n",
    "for X, y in test_dataloader:\r\n",
    "    print(\"Shape of X [N, C, H, W]: \", X.shape)\r\n",
    "    print(\"Shape of y: \", y.shape, y.dtype)\r\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more about `loading data in PyTorch <data_tutorial.html>`_.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating Models\n",
    "------------------\n",
    "To define a neural network in PyTorch, we create a class that inherits\n",
    "from `nn.Module <https://pytorch.org/docs/stable/generated/torch.nn.Module.html>`_. We define the layers of the network\n",
    "in the ``__init__`` function and specify how data will pass through the network in the ``forward`` function. To accelerate\n",
    "operations in the neural network, we move it to the GPU if available.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n",
      "NeuralNetwork(\n",
      "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
      "  (linear_relu_stack): Sequential(\n",
      "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
      "    (5): ReLU()\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Get cpu or gpu device for training.\r\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\r\n",
    "print(\"Using {} device\".format(device))\r\n",
    "\r\n",
    "# Define model\r\n",
    "class NeuralNetwork(nn.Module):\r\n",
    "    def __init__(self):\r\n",
    "        super(NeuralNetwork, self).__init__()\r\n",
    "        self.flatten = nn.Flatten()\r\n",
    "        self.linear_relu_stack = nn.Sequential(\r\n",
    "            nn.Linear(28*28, 512),\r\n",
    "            nn.ReLU(),\r\n",
    "            nn.Linear(512, 512),\r\n",
    "            nn.ReLU(),\r\n",
    "            nn.Linear(512, 10),\r\n",
    "            nn.ReLU()\r\n",
    "        )\r\n",
    "\r\n",
    "    def forward(self, x):\r\n",
    "        x = self.flatten(x)\r\n",
    "        logits = self.linear_relu_stack(x)\r\n",
    "        return logits\r\n",
    "\r\n",
    "model = NeuralNetwork().to(device)\r\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more about `building neural networks in PyTorch <buildmodel_tutorial.html>`_.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optimizing the Model Parameters\n",
    "----------------------------------------\n",
    "To train a model, we need a `loss function <https://pytorch.org/docs/stable/nn.html#loss-functions>`_\n",
    "and an `optimizer <https://pytorch.org/docs/stable/optim.html>`_.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "loss_fn = nn.CrossEntropyLoss()\r\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a single training loop, the model makes predictions on the training dataset (fed to it in batches), and\n",
    "backpropagates the prediction error to adjust the model's parameters.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def train(dataloader, model, loss_fn, optimizer):\r\n",
    "    size = len(dataloader.dataset)\r\n",
    "    model.train()\r\n",
    "    for batch, (X, y) in enumerate(dataloader):\r\n",
    "        X, y = X.to(device), y.to(device)\r\n",
    "\r\n",
    "        # Compute prediction error\r\n",
    "        pred = model(X)\r\n",
    "        loss = loss_fn(pred, y)\r\n",
    "\r\n",
    "        # Backpropagation\r\n",
    "        optimizer.zero_grad()\r\n",
    "        loss.backward()\r\n",
    "        optimizer.step()\r\n",
    "\r\n",
    "        if batch % 100 == 0:\r\n",
    "            loss, current = loss.item(), batch * len(X)\r\n",
    "            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also check the model's performance against the test dataset to ensure it is learning.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def test(dataloader, model, loss_fn):\r\n",
    "    size = len(dataloader.dataset)\r\n",
    "    num_batches = len(dataloader)\r\n",
    "    model.eval()\r\n",
    "    test_loss, correct = 0, 0\r\n",
    "    with torch.no_grad():\r\n",
    "        for X, y in dataloader:\r\n",
    "            X, y = X.to(device), y.to(device)\r\n",
    "            pred = model(X)\r\n",
    "            test_loss += loss_fn(pred, y).item()\r\n",
    "            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\r\n",
    "    test_loss /= num_batches\r\n",
    "    correct /= size\r\n",
    "    print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training process is conducted over several iterations (*epochs*). During each epoch, the model learns\n",
    "parameters to make better predictions. We print the model's accuracy and loss at each epoch; we'd like to see the\n",
    "accuracy increase and the loss decrease with every epoch.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1\n",
      "-------------------------------\n",
      "loss: 2.294390  [    0/60000]\n",
      "loss: 2.296019  [ 6400/60000]\n",
      "loss: 2.284610  [12800/60000]\n",
      "loss: 2.289925  [19200/60000]\n",
      "loss: 2.272185  [25600/60000]\n",
      "loss: 2.263884  [32000/60000]\n",
      "loss: 2.260055  [38400/60000]\n",
      "loss: 2.248945  [44800/60000]\n",
      "loss: 2.256240  [51200/60000]\n",
      "loss: 2.219766  [57600/60000]\n",
      "Test Error: \n",
      " Accuracy: 30.0%, Avg loss: 2.240785 \n",
      "\n",
      "Epoch 2\n",
      "-------------------------------\n",
      "loss: 2.238276  [    0/60000]\n",
      "loss: 2.253145  [ 6400/60000]\n",
      "loss: 2.222885  [12800/60000]\n",
      "loss: 2.234439  [19200/60000]\n",
      "loss: 2.196337  [25600/60000]\n",
      "loss: 2.163118  [32000/60000]\n",
      "loss: 2.178855  [38400/60000]\n",
      "loss: 2.149394  [44800/60000]\n",
      "loss: 2.174822  [51200/60000]\n",
      "loss: 2.074628  [57600/60000]\n",
      "Test Error: \n",
      " Accuracy: 39.9%, Avg loss: 2.134625 \n",
      "\n",
      "Epoch 3\n",
      "-------------------------------\n",
      "loss: 2.152172  [    0/60000]\n",
      "loss: 2.176278  [ 6400/60000]\n",
      "loss: 2.110262  [12800/60000]\n",
      "loss: 2.123178  [19200/60000]\n",
      "loss: 2.048111  [25600/60000]\n",
      "loss: 2.001813  [32000/60000]\n",
      "loss: 2.028304  [38400/60000]\n",
      "loss: 1.982752  [44800/60000]\n",
      "loss: 2.046253  [51200/60000]\n",
      "loss: 1.850700  [57600/60000]\n",
      "Test Error: \n",
      " Accuracy: 41.0%, Avg loss: 1.974920 \n",
      "\n",
      "Epoch 4\n",
      "-------------------------------\n",
      "loss: 2.022519  [    0/60000]\n",
      "loss: 2.059759  [ 6400/60000]\n",
      "loss: 1.955057  [12800/60000]\n",
      "loss: 1.971380  [19200/60000]\n",
      "loss: 1.815324  [25600/60000]\n",
      "loss: 1.803387  [32000/60000]\n",
      "loss: 1.808985  [38400/60000]\n",
      "loss: 1.760346  [44800/60000]\n",
      "loss: 1.851667  [51200/60000]\n",
      "loss: 1.596924  [57600/60000]\n",
      "Test Error: \n",
      " Accuracy: 44.9%, Avg loss: 1.771660 \n",
      "\n",
      "Epoch 5\n",
      "-------------------------------\n",
      "loss: 1.832917  [    0/60000]\n",
      "loss: 1.901475  [ 6400/60000]\n",
      "loss: 1.809968  [12800/60000]\n",
      "loss: 1.830599  [19200/60000]\n",
      "loss: 1.582441  [25600/60000]\n",
      "loss: 1.643260  [32000/60000]\n",
      "loss: 1.616675  [38400/60000]\n",
      "loss: 1.587723  [44800/60000]\n",
      "loss: 1.699573  [51200/60000]\n",
      "loss: 1.410080  [57600/60000]\n",
      "Test Error: \n",
      " Accuracy: 48.9%, Avg loss: 1.626094 \n",
      "\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "epochs = 5\r\n",
    "for t in range(epochs):\r\n",
    "    print(f\"Epoch {t+1}\\n-------------------------------\")\r\n",
    "    train(train_dataloader, model, loss_fn, optimizer)\r\n",
    "    test(test_dataloader, model, loss_fn)\r\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more about `Training your model <optimization_tutorial.html>`_.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving Models\n",
    "-------------\n",
    "A common way to save a model is to serialize the internal state dictionary (containing the model parameters).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved PyTorch Model State to model.pth\n"
     ]
    }
   ],
   "source": [
    "torch.save(model.state_dict(), \"model.pth\")\r\n",
    "print(\"Saved PyTorch Model State to model.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading Models\n",
    "----------------------------\n",
    "\n",
    "The process for loading a model includes re-creating the model structure and loading\n",
    "the state dictionary into it.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = NeuralNetwork()\r\n",
    "model.load_state_dict(torch.load(\"model.pth\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model can now be used to make predictions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n"
     ]
    }
   ],
   "source": [
    "classes = [\r\n",
    "    \"T-shirt/top\",\r\n",
    "    \"Trouser\",\r\n",
    "    \"Pullover\",\r\n",
    "    \"Dress\",\r\n",
    "    \"Coat\",\r\n",
    "    \"Sandal\",\r\n",
    "    \"Shirt\",\r\n",
    "    \"Sneaker\",\r\n",
    "    \"Bag\",\r\n",
    "    \"Ankle boot\",\r\n",
    "]\r\n",
    "\r\n",
    "model.eval()\r\n",
    "x, y = test_data[0][0], test_data[0][1]\r\n",
    "with torch.no_grad():\r\n",
    "    pred = model(x)\r\n",
    "    predicted, actual = classes[pred[0].argmax(0)], classes[y]\r\n",
    "    print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more about `Saving & Loading your model <saveloadrun_tutorial.html>`_.\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "1baa965d5efe3ac65b79dfc60c0d706280b1da80fedb7760faf2759126c4f253"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
